Hi @wu-sheng ,
I’ve implemented the duplicate data verification feature as described in issue #12253.
The PR adds a CheckForDuplicates() utility along with corresponding unit tests.

It would be great if you could please review it when you have a moment.
I’d also be happy to extend this in a follow-up to integrate with the existing list verification flow if needed.

This PR is ready for review and merge.
It adds the duplicate data verification utility (CheckForDuplicates) with unit tests as discussed in issue #12253.

All tests have passed locally, and the change is backward-compatible.

Hi @kezhenxu94, thanks for the feedback!

I'd be happy to expose this to YAML. Could you clarify the preferred integration approach?

Option A:-Add a noDuplicates: true field to existing list verifications in YAML
Option B:- Create a new verification type specifically for duplicate checking

I'm leaning towards Option A for better integration with existing workflows. Would that work, or do you have a different pattern in mind?

Happy to implement whichever approach fits the project best!

@Navadeep0007 option A looks good to me, please go ahead, thanks!

Hi @kezhenxu94,
I’ve implemented the YAML integration for Option A (noDuplicates: true) as discussed.
The flag is now supported in list verifications and calls CheckForDuplicates().
Tests and examples have been updated.

Added missing Apache License headers to the following files to fix the CI failure reported by license-eye:

examples/list_verification.yaml

internal/verify/config.go

internal/verify/runner.go

The build should now pass successfully. ✅